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CONTENT roENTIFIERS TRIGGERING CORRESPONDING RESPONSES 


Related Application Data 
This application is a/continuation in part of application 09/571,422, which 
i priority from aMmcations 09/314,648, 09/342,688, 09/342,689, 09/342,971, 
09/343,101, 09/343,10^ 60/141,468, 60/151,586, 60/158,015, 60/163,332, 60/164,619, 
09/531,076, 09/543/125, 09/547,664, and 09/552,998. 

This appHcation is also a continuation-in-part of copending applications 
09/574,726 and 09/476,686, both of which claim priority to apphcation 60/134,782. 
The present application claims priority benefit to the foregoing appUcations. 
The subject matter of this application is also related to that of 09/620,019, 
60/257,822, 60/232,163, and 09/404,291. 

Field of the Invention 
The present invention relates to computer-based systems, and more particularly 
relates to systems that identify electronic or physical objects (e.g., audio, printed 
documents, video, etc.), and trigger corresponding responses. 

Background 

In appHcation 09/571,422 (now laid-open as PCT publication WO 00/70585), the 
present assignee described technology that can sense an object identifier from a physical 
or electronic object, and trigger a corresponding computer response. 

In applications 09/574,726 and 09/476,686, the present assignee described 
technology that uses a microphone to sense audio soimds, determine an identifier 
corresponding to the audio, and then trigger a corresponding response. 

Detailed Description 

Although the cited patent applications focused on use of digital watermarks to 
identify the subject objects/audio, they noted that the same applications and benefits can 
be provided with other identification technologies. 

One such suitable technology - variously known as robust hashing, fingerprinting. 
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etc. - involves generating an identifier fi^om attributes of the content. This identifier can 

then be looked-up in a database (or other data structure) to determine the song (or other 

audio track) to which it corresponds. 

Various fingerprinting technologies are known. For example, a software program 

called TRM, fi^om Relatable Software, was written up in the Washington Post as follows: 
TRM performs a small technological miracle: It "fingerprints*' songs, 
analyzing beat and tempo to generate a unique digital identifier Since 
every song is slightly different, no two "acoustic fingerprints" are alike, 
not even live and studio versions of the same melody. 

Tuneprint is another such audio fingerprinting tool. Tuneprint is understood to 
utilize a model of human hearing used to predict how audio will appear after it's been 
distorted by the human ear, and the parts of neural processing that are imderstood. This is 
O some of the same information that led to MP3 encoders achieving exceptional audio 
compression. Characteristics that uniquely identify the track are then identified by 

m 

picking out the most important, surprising, or significant features of the soimd. 
H' Yet another fingerprinting program is Songprint, available as an open source 

^ library fi"om fi:'eetantrum.org. 

One form of fingerprint may be derived by applying content - in whole or part, 

Q 

in and represented in time- or fi^equency format - to a neural network, such as a Kohonen 

P self-organizing map. For example, a song may be identified by feeding the first 30 

O seconds of audio, with 20 millisecond fourier transformed windows, into a Kohonen 

Li 

network having 64 outputs. The 64 outputs can, themselves, form the fingerprint, or they 
can be fiirther processed to yield the fingerprint. 

"""^^A-i^ariety of other fingerprinting tools and techniques are known to artisans in the 
field. Others ^e^iSclo^ed, e.g., in applications 60/257,822, 09/563,664, and 09/578,551. 
See also the chapter on Fingerprintiiig by John Hyeon Lee, in Information Hiding: 
Techniques for Steganography and DigitaTWatemiarking edited by Stefan Katzenbeisse 
and Fabien A.P. Petitcolas, pubhshed by Artech Ho^lse^ 

One way to generate a fingerprint is to "hash" the audio, to derive a shorter code 
that is dependent, in a predetermined way, on the audio data. However, slight differences 
in the audio data (such as sampling rate) can cause two versions of the same song to yield 
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two different hash codes. While this outcome is advantageous in certain outcomes, it is 
disadvantageous in many others. 

Generally preferable are audio fingerprinting techniques that yield the same 
fingerprints, even if the audio data are slightly different. Thus, a song sampled at a 96K 
bitrate desirably should yield the same fingerprint as the same song sampled at 128K. 
Likewise, a song embedded with steganographic watermark data should generally yield 
the same fingerprint as the same song without embedded watermark data. 

One way to do this is to employ a hash function that is insensitive to certain 
changes in the input data. Thus, two audio tracks that are acoustically similar will hash to 
the same code, notwithstanding the fact that individual bits are different. A variety of 
such hashing techniques are known. 

Another approach does not rely on "hashing" of the audio data bits. Instead, the 
audio is decomposed into elements having greater or lesser perceptibility. Audio 
compression techniques employ such decomposition methods, and discard the elements 
that are essentially imperceptible. Li fingerprinting, these elements can also be 
disregarded, and the "fingerprint" taken fi-om the acoustically significant portions of the 
audio (e.g., the most significant coefficients after transformation of the audio into a 
transform domain, such as DCT). 

Some fingerprinting techniques do not rely on the absolute audio data (or 
transformed data) per se, but rather rely on the changes in such data firom sample to 
sample (or coefficient to coefficient) as an identifying hallmark of the audio. 

Some fingerprinting algorithms consider the entire audio track (e.g., 3 minutes). 
Others work on much shorter windows - a few seconds, or fi-actions of seconds. The 
former technique yields a single fingerprint for the track. The latter yields plural 
fingerprints - one from each excerpt. (The latter fingerprints can be concatenated, or 
otherwise combined, to yield a master fmgerprint for the entire audio track.) For 
compressed audio, one convenient unit fi'om which excerpts can be formed is the fi'ame 
or window used in the compression algorithm (e.g., the excerpt can be one firame, five 
fi-ames, etc.). 

One advantage to the excerpt-based techniques is that a song can be correctly 
identified even if it is truncated. Moreover, the technique is well suited for use with 
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streaming media (in which the entire song data is typically not available all at once as a 
single file). 

In database look-up systems employing fingerprints fi-om short excerpts, a first 
fingerprint may be found to match 10 songs. To resolve this ambiguity, subsequent 
excerpt-fingerprints can be checked. 

One way of making fingerprints "robust" against variations among similar tracks 
is to employ probabilistic methods using excerpt-based fingerprints. Consider the 
following, over-simplified, example: 


Fingerprinted excerpt 

Matches these songs in database 

Fingerprint 1 

A,B,C 

Fingerprint 2 

C, D,E 

Fingerprint 3 

B,D,F 

Fingerprint 4 

B,F,G 


In this situation, it appears most probable that the fingerprints correspond to song 
B, since three of the four excerpt-fingerprints support such a conclusion. (Note that one 
of the excerpts - that which yielded Fingerprint 2 - does not match song B at all.) 

More sophisticated probabilistic techniques, of course, can be used. 

Once a song has been identified in a database, a number of different responses can 
be triggered. One is to impose a set of usage controls corresponding to terms set by the 
copyright holder (e.g., play control limitations, record control, fee charges, etc.) Another 
is to identify metadata related to the song, and provide the metadata to a user (or a link to 
the metadata). In some such applications, the song is simply identified by title and artist, 
and this information is retumed to the user, e.g., by email, instant messaging, etc. With 
this information, the user can be given an option to purchase the music in CD or 
electronic form, purchase related materials (t-shirts, concert tickets), etc. A great variety 
of other content-triggered actions are disclosed in the cited applications. 

One of the advantages of fingerprint-based content identification systems is that 
they do not require any alteration to the content. Thus, recordings made 50 years ago can 
be fingerprinted, and identified through such techniques. 

Going forward, there are various advantages to encoding the content with the 
fingerprint. Thus, for example, a fingerprint identifier derived fi-om a song can be stored 


WYC:lmp 5/14/01 P03 


-5- 


E^^HSS MAIL NO. ELS 1 82667 17US 


in a file header of a file containing that song. (MP3 files, MPEG files, and most other 
common content file formats include header fields in which such information can readily 
be stored.) The fingerprint can then obtained in two different ways - by reading the 
header info, and by computation fi"om the audio information. This redimdancy offers 
several advantages. One aids security. If a file has a header-stored fingerprint that does 
not match a fingerprint derived fi-om the file contents, something is amiss - the file may 
be destructive (e.g., a bomb or virus), or the file structure may mis-identify the file 
contents. 

hi some embodiments, the fingerprint data (or watermark data) stored in the 
header may be encrypted, and/or authenticated by a digital signature such as a complete 
hash, or a few check bits or CRC bits. In such cases, the header data can be the primary 
source of the fingerprint (watermark) information, with the file contents being processed 
to re-derive the fingerprint (watermark) only if authentication of the fingerprint stored in 
the header fails. Instead of including the fingerprint in the header, the header can include 
an electronic address or pointer data indicating another location (e.g., a URL or database 
record) at which the fingerprint data is stored. Again, this information may be secured 
using known techniques. 

Similarly, the fingerprint can point to a database that contains one or more IDs 
that are added via a watermark. This is useful when CDs are being converted to MPS 
files (i.e. ripped) and the fingerprint is calculated firom a hash of the table of contents 
(TOC) such as done with CDDB.com, or from all of the songs. In this case, the database 
entry for that fingerprint could include a list of IDs for each song, and these IDs are 
added via a watermark and/or fi-ame header data. This can also be useful where the 
content is identified based upon a group of fingerprints fi-om plural excerpts, in which 
case the database that determines the content also contains an identifier, unrelated to the 
fingerprint(s) for that piece of content that can be embedded via a watermark. 

^instead of, or in addition to, storing a fingerprint in a file header, the fingerprint 
data may be stegaitogcaphically encoded into the file contents itself, using known 
watermarking techniques (e?^:>4hose disclosed in appHcation 09/503,881, and patents 
6,061,793, 6,005,501 and 5,940,1 3 5)>>:^ example, the fingerprint ID can be duplicated 
in the data embedded via a watermark. 
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In some arrangements, a watermark can convey a fingerprint, and auxiliary data 
as well. The file header can also convey the fingerprint, and the auxiliary data. And even 
if the file contents are separated from the header, and the watermark is corrupted or 
otherwise lost, the fingerprint can still be recovered fi*om the content. In some cases, the 
lost axixiliary data can altematively be obtained fi'om information in a database record 
identified by the fingerprint (e.g., the auxiliary information can be literally stored in the 
record, or the record can point to another source where the information is stored). 

Instead of especially processing a content file for the purpose of encoding 
fingerprint data, this action can be done automatically each time certain applications 
process the content for other purposes. For example, a rendering application (such as an 
MPS player or MPEG viewer), a compression program, an operating system file 
management program, or other-purposed software, can calculate the fingerprint fi^om the 
content, and encode the content with that information (e.g., using header data, or digital 
^ watermarking). It does this while the file is being processed for another purpose, e.g., 

taking advantage of the file's copying into a processing system's RAM memory, fi-om 
H slower storage. 

,J In formats in which content is segregated into portions, such as MPS fi'ames, a 

1_ fingerprint can be calculated for, and encoded in association with, each portion. Such 

O 

m fingerprints can later be cross-checked against fingerprint data calculated firom the 

^ content information, e.g., to confirm delivery of paid-for content. Such fingerprints may 

be encrypted and locked to the content, as contemplated in application 09/620,019. 

addition, in this fi-ame based systems, the fingerprint data and/or watermark 
data can be embeSdecJwith some or all data throughout each firames. This way a 
streaming system can use mSiae^der to first check the song for identification, and if that 
identification is absent or not authenticaJSdjs^ system can check for the watermark 
and/or calculate the fingerprint. This improves the^ffici^ncy and cost of the detecting 
system. 

Before being encrypted and digitally signed, the data in the fi*ame header can be 
modified by the content, possibly a hash of the content or a few critical bits of content. 
Thus, the fi-ame header data cannot be transferred between content. When reading the 
data, it must be modified by the inverse transform of the earlier modification. This 


WYC:lmp 5/14/01 P037( 



-7- 


e: 



Is MAIL NO. ELS 1 82667 1 7US 


system can be applied whether the data is embedded throughout each frame or all in a 
global file header and is discussed in appHcation serial 09/404,291 entitled "Method And 
Apparatus For Robust Embedded Data" by Ken Levy on 9/23/99. Reading this secure 
header data is only slightly more complex than without the modification, such that the 
system is more efficient than always having to calculate the fingerprint and/or detect the 
watermark. 

To provide a comprehensive disclosure without unduly lengthening this 
specification, applicants incorporate by reference the patents and patent applications cited 
above. It is applicant's express intention to teach that the methods detailed herein are 
applicable in connection with the technologies and appUcations detailed in these cited 
patents and applications. 

Although the foregoing specification has focused on audio applications, it will be 
recognized that the same principles are likewise applicable with other forms of content, 
including still imagery, motion pictures, video, etc. Thus, for example, Digimarc 
MediaBridge linking from objects to corresponding intemet resources can be based on 
identifiers derived from captured image data or the like, rather than from embedded 
watermarks. As such, the technique is applicable to images and video. 


